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ABSTRACT 

Problems are frequently encountered when attempting 
to improve the evaluation design of an ongoing educational program 
which has not made adequate provisions for evaluation. Usually the 
objectives of such a program are too global to guide the evaluator in 
the development of a more functional evaluation design. This paper 
presents a discussion of the activity analysis approach which is 
helpful in such a situation to obtain information needed in the 
development of an improved evaluation design. Once this information 
has been obtained it can be used by an evaluator in the development 
of a more comprehension evaluation design for the ongoing program. 
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INTRODUCTION 



Although most state and federal agencies require that an 
evaluation design accompany a proposal for the funding of a 
new educational program, some local school districts, community 
agencies, and regional centers, are receiving funds to support 
programs which do not have adequate provisions for evaluation. 
As administrators of such programs become aware that .the amount 
of future funding is dependent on the quality of the evaluation 
of their project, they seek consultant assistance for the de- 
velopment of more substantive evaluation designs. 

A recent event which precipitated such activity was the 

v — 

transfer of some ongoing educational programs from the United 
States Office of Education and the Office of Educational 
Opportunity to the National Institute of Education. Some of 
these ongoing projects were funded on the basis of the needs 
they would service and the unique means in which they would 
meet those needs. For some projects, the evaluation design 
consisted of the minimal assessment of summative outcomes. 
Since NIE has required a more comprehensive evaluation of their 
funded programs, many project administrators were faced with 
incorporating formative evaluation activities into their designs 
as well as improving the quality of their summative evaluation 
activities. While working with program administrators confront- 
ed with the task of improving the evaluation component of their 



project, it has become evident that the major problem in this 
effort is communication. More specifically, the quality of the 
new evaluation design as well as the time needed to develop 
such . design is dependent upon the ability of the program 
staft to communicate the essence of the project to the evalua- 
tion consultant. The activity analysis approach has been devel- 
oped to maximize communication between the program staff and 
evaluation consultant and thus, maximize their efforts to 
develop a more functional design for the evaluation of the 
project. 

'-■< 

BACKGROUND 

The major problem encountered when attempting to improve 
the evaluation design of an ongoing program is usually the 
failure of the program objectives to communicate to the eva- 

i 

luator the specific activities being undertaken or the specific 
outcomes expected. Often the evaluator is presented with pro- 
gram objectives which are similar to the one which follows: 

Through the implementation of a Comprehensive 
Diagnostic Center, Project Help will expedite 
the referral and diagnosis of students with 
learning disabilities. 

Although this objective conveys the general activity of esta- 
blishing a diagnostic center and the general outcomes of expe- 
diting the referral and diagnosis of students with learning dis 
abilities, this information is not sufficient for the develop- 
ment of a functional evaluation^ design, that is, an evaluation 



design which would provile the program administrator with both 
formative and summative feedback* In order to obtain the more 
specific information desired, the evaluator has tended tradi- 
tionally to review the documentation on the program and meet 
with program personnel until enough specific information has 
been collected to formulate an appropriate evaluation design. 
Although this practice is widely accepted, some problems can be 
encountered when applying this approach in a program setting. 

First, the process of meeting with program staff to clarify 
activities and outcomes can be quite Costly, especially for 
large projects, in some large program settings, it could take 
up to two weeks of on-site visitation for a consultant to obtain 
the necessary information to develop a functional evaluation 
design. Often programs in need of this assistance, do not have 
the necessary funds at their disposal to support such preliminary 
evaluation activities. 

Secondly, it is not always easy to obtain the assistance 
of a competent evaluation specialist to improve the evaluation 
of an ongoing program. Some evaluation specialists do not look 
forward to meeting with program personnel to extract from them 
the specifics of their project activities and intents. Their 
basic reason being the amount of time "wasted" in the process 
of general introductions to program personnel, coffee breaks, 
telephone interruptions, and discussions of the generalities 
of the program before one gets down to the real specifics of 
the program. Many qualified evaluation specialists find it 



4 

discouraging to work in such practical settings since they 
feel it is an inefficient use of their time and expertise. 

Finally; in cases where the program objectives as well 
as other documentation are vague, meetings with program per- 
sonnel for further clarification of program activities and 
intents can focus on those aspects of the project which the 
staff fljould like to see evaluated rather than those aspects 
which should be evaluated. A program evaluation design based 

* 

on such input would not be comprehensive, but instead, would 
focus on those areas of crucial interest to the program staff. 
Although some evaluation specialists might argue that this 
practice is acceptable, most funding agencies would disagree. 
The goal of -thr evaluator is to take a comprehensive look at 
the impact of the total program, not just those areas of pri- 
mary concern to the program staff. 

Although these are not all of the problems which can be 
encountered when the evaluation specialist and program staff 
join forces to improve the evaluation design of an ongoing 
program, they are some of the more common difficulties exper- 
ienced. These problems have not been cited to discourage such 
joint efforts, but rather to illustrate the need to develop a 
less cfcstly and more efficient process whereby the evaluation 
specialist and program staff can work together toward their 
common goal of developing a more comprehensive design for the 
evaluation of an ongoing educational program. Activity ana- 
lysis was developed in response to this need. 
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THE ACTIVITY ANALYSIS RATIONALE 

Activity analysis is an approach to the development of 
an improved evaluation design through the systematic delinea- 
tion and analysis of the activities inherent in each of the 
global objectives of an ongoing program. Although activity 
analysis can be introduced into an ongoing program by an 
evaluation specialist, the primary responsibility for the 
review of activities related to program objectives rests 
with the program personnel. Initially, activity analysis may 
appear to be a new term for the restatement of global objec- 
tives in specific operational terms. As the rationale for 
activity analysis is presented, it will become evident that 
the restatement of global objectives in operational terms is 
only one aspect of activity analysis. 

The initial step in applying the activity analysis ra- 
tionale to an ongoing program is to delineate the activities 
inherent in each of the global objectives of the program. 
When listing these activities, they should be described in 
operational terms. Next, it is important to evaluate the 
activities for each objective on the basis of their clarity, 
necessity, and sufficiency relative to the attainment of the 
objective. In other words, for each objective it is necessary 
to review the activities listed to be sure they are described 
clearly in operational terms which would convey the essence 
of the activity to a person not closely involved with the 
operation of the program. Also, it is important to assess 
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the activities related to each objective as a whole to deter-* 
mine whether it is necessary to pursue each activity in order 
to attain the objective. Peripheral activities which are 
not essential to the attainment of the objective may be eli- 
minated. Finally, the activities directed toward each pro* 
gram objective should be examined in total to determine 
whether the completion of these activities would insure the 
attainment of the program objective,. If the activities are 
not sufficient for the achievement of a program objective , 
it is necessary to delineate those additional activities which 
must be pursued in order to reach the objective. 

Once the activities related to each program objective have 
been adequately delineated and screened according to the cri- 
teria of clarity, necessity, and sufficiency, the structure 
and sequence of these activities should be specified. The 
structure consists of a detailed explanation of each activity, 
whereas the sequence is the temporal arrangement of these acti- 
vities. Once the structure and sequence of the activities 
related to each objective are specified, the program adminis- 
trator has at his disposal a detailed blueprint or map of the 
program objectives and concoramitent activities which the pro- 
ject proposes to complete. The general format of this map is 
illustrated in Figure 1. 

Before proceeding with the activity analysis rationale 
the program administrator should conduct an objective by ob- 
jective review of this map to determine whether the resources 
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FIGURE 1 



General Format of the Map of Program Objectives and _ ... 
Activities Developed Through the Application of Activity Analysis 



PROGRAM OBJECTIVE 1 
Activity 11 

- Detailed description of activity 11 
Activity 12 

- Detailed description of activity 12 
Activity 13 

- Detailed description of activity 13 



Activity In 

- Detailed description of activity In 

PROGRAM OBJECTIVE 2 
Activity 21 

- Detailed description of activity 21 
Activity 22 

- Detailed description of activity 22 
Activity 23 

- Detailed description of activity 23 



Activity 2n 

- Detailed description of activity 2n 



PROGRAM OBJECTIVE N 
Activity Nl 

- Detailed description of activity Nl 
Activity N2 

- Detailed description of activity N2 
Activitv N3 

- Detailed description of activity N3 



Activity Nn 

- Detailed description of activity Nn 
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o£ the program are sufficient to complete an effort of this 
magnitude, if the necessary program resources are not availa- 
ble/ decisions must be made to reduce the scope of some of 
the activities, eliminate some activities entirely, and 
possibly, to delete some entire objectives from the program 
plan. The result of such a review is a revised map of pro- 
gram objectives and concoromitent activites which can be com- 
pleted realistically, given the available program resources. 

The final step in applying the activity analysis rationale 
consists of developing an activity analysis table for each pro- 
gram objective on the basis of the information presented in 
the program map. A sample activity analysis table is presented 
in Table 1. An activity analysis table is similar to the 
program map in that it contains a statement of the program 
objective as well as a delineation of the program activities 
essential to the attainment of that objective. In addition, 
the activity analysis table includes such information as: the 
agents responsible for the activity, starting date, planned 
completion date, actual completion date, and activity score. 
Responsibility for an activity can be either primary or secon- 
dary. If an agent assumes primary responsibility, the name 
of this agent appears in capital letters. However, if the 
agent assumes secondary responsibility, the name of this 
agent appears in lower case letters. The activity score is 
a subjective estimate of the relative contribution of the 
activity to the attainment of the objective stated in the 
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TABLE 1 

r 

Sample Activity Analysis Table 
STATEMENT OP THE GLOBAL PROGRAM OBJECTIVE 



Activity 



JUL 



I 



_£Q_ 



I 0> 
S +> 

II 

(3 v 
Ck. a 



I 4J 



-w 



I 

1 



Activity 1 

- description 

Activity 2 

- description 

Activity 3 

- description 

Activity 4 

- description 

Activity 5 

- description 



PRIMARY 
Secondary 

PRIMARY 

Secondary 

PRIMARY 
Secondary 

PRIMARY 
Secondary 

PRIMARY 
Secondary 



7/72 
7/73 
9/73 
9/73 
1/74 



7/73 
10/73 
10/74 
10/74 

6/74 



6/73 
11/73 



15 
5 
45 



10 



25 



Total Activity Score (TS) = 100 

Sum of the activity scores for the completed activities 
(SAC) * 20 

% Completed to date for this program objective - 20% 
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table. This estimate is made by the program director clfter 
taking into consideration the amount of time and other program 
resources which must be expended in order to complete the ac- 
tivity. It is evident from the sample activity analysis 
table that: 
IP 

SAC ■ Sum of the activity scores of the completed 
activities 
TS = Total activity score 

THEN 

x 100 • % completion to date for the program 
Tb objective 

The calculation of percent completion to date is simplified 
somewhat if the scores assigned to the activites in the table 
are based on a total activity score of one hundred (100). In 
such cases, percent completion to date is equal to the sum 
of the scores for the activities completed. 

Once the activity analysis rationale has been applied to 
the global objectives of an ongoing program, the program staff 
can better communicate their intents and activities to an 
evaluation consultant. The resultant activity analysis tables 
provide the evaluation consultant with the following informa- 
tion regarding each global objective of the program? 

1. A description of each program activity in 
operational terms; 

2. An indication of the individual or agent 
responsible for each activity? 

3. An indication of when each program activity 
has been or will be initiated; 

4. An indication of when each program activity 
has been or v/ill be completed. 



11 



Given this type of information, the evaluation specialist can 
work efficiently with program staff on the development of an 
improved design for the evaluation of the program. Although 
some time will still be spent with program staff on clarifying 
some of the program activities and intents conveyed through 
the activity analysis tables, the time spent with program 
staff is significantly reduced throuch the application of the 
activity analysis rationale. 

In addition to facilitating the development of an im- 
proved evaluation design, the total program benefits from the 
application of activity analysis since the program director is 
forced to develop a detailed description aridisdhedule of pro- 
gram activities which can be attained realistically. The 
format of this program description and schedule (i.e., the 
activity analysis tables) enables the program director to 
estimate at any time the progress of the program toward the 
attainment of its objectives. 

CONCLUDING REMARKS 

The activity analysis approach has been used by the 
author in the development of an evaluation design for several 
programs. The cooperation of program directors in the appli- 
cation of activity analysis has been quite favorable. The 
largest program to which activity analysis has been applied 
is the Holliston (Massachusetts) Middle School Project 
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(Title III, ESEA) * . In one month the activity analysis ra- 
tionale was applied to the twenty-three major objectives of 
the Middle School Project. An activity analysis table for 
one of the objectives of this project is presented in Table 2. 
In this particular application of activity analysis the des- 
cription of each activity related to this major program objec- 
tive was placed in a narrative section following the activity 
analysis table, rather than in the table itself. 

In addition to facilitating the development of an im- 
proved evaluation design, activity analysis provides program 
staff with a sense of involvement in the formulation of this 
design. When utilizing the traditional approach of meeting 
with program staff individually to clarify program activities 
and intends, the staff does not usually perceive their comments 
to be direct inputs into the formulation of an- improved 
evaluation design. Sitting down and talking with an evalua- 
tion consultant does not tend to give one the feeling of 
actively participating in the evaluation design process. On 
the other hand, if staff are familiarized with the activity 
analysis rationale and asked to develop activity analysis 
tables, they tend to feel they are making a tangible contribu- 
tion to the process of formulating an improved evaluation de- 
sign. This feeling of active participation has added benefits 

4 

| | „ | | „ | ■ || ^ 

*Holliston's Middle School Project* A Systems Approach 
to Middle School Education. Application for Continuation 
Grant, November 30, 1968, Title III, ESEA Project No. 68-944. 
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TABLE 2 



An Example of the Application of Activity Analysis to One of 
The Major Objectives of the Holliston Middle School Project 



TO PILOT TEST , REFINE, AND E XTEND THE KNOWLEDGE CONTROL INVENTORY (KCI) WHICH 
HAS BEEN DEVELOPED IN PROTOTYPE FORM DURING THE PLANNING PHASE OF THE PROJECT 



THE DESIGN PHASE 



a 

1 4jj 

, w, at 



t 



1 

JL 



i 



THE DESIGN ACTIVITY 



1. 


To Research and Develop a prototype 


PROJECT 1 








15 




a. 


In Language Arts 


School 


2/68 


6/68 


6/68 




b. 


In Social Studies 












2. 


To Pilot Tent prototype in clinical 


PROJECT 








15 




summer school during planning cycle 


School 


7/68 


8/68 


8/68 




a. 


In Language Arts 














b. 


In Social Studies 












3. 


To PiJ.ot Test prototype in clinical 


PROJECT 








15 




summer school during teaching cycle 


School 


7/68 


8/68 


8/68 




a. 


In Language Arts 














b. 


In Social Studies 












4. 


To Refine social studies prototype 


PROJECT 








15 








School 


9/68 


4/69 


4/69 


5. 


To Extend KCI into each curriculum 


PROJECT 












area 


School 








6.67 




a. 


Into Social Studies 




9/68 


4/69 


4/69 




b. 


Into Reading 




9/68 


4/69 


4/69 


6.67 




c. 


Into Language Arts 
















(1) Sentence Study 




9/68 


4/69 


4/69 


1.67 






(2) Word Study 




9/68 


4/69 




1.67 






(3) Literature 




3/68 


1/69 




1.67 






(4) Composition 




2/70 


6/70 


8/69 


1« 67 




d. 


Into Mathematics 




7/69 


8/69 


6.67 




e 


Into Science 




7/69 


8/69 


8/69 


6.67 




f. 


Into Unified Arts 




7/69 


1/70 




6.67 



— PERCENT COMPLETE: 11/30/69*88% 

1. The upper case letters indicate primary responsibility 
The lower case letters indicate secondary responsibility 



100.00 
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since program staff tend to be more receptive to cooperating 
in the implementation of a design based on their contributions. 
A final strength of the activity analysis rationale is 
-) that it provides a measure of the status of implementation of 
a program which can be reported with the program evaluation 
findings. Many funding agencies are concerned that the eva- 
luation results reported by some programs are not consistent 
with expectations since the programs have not been implemented 
sufficiently. For example, a vocational education program 
may not have achieved its objective of increasing the occupa- 
tional awareness of program participants. This outcome can 
be accounted for in one of two ways . Either the program was 
not successful in bringing about the desired change in the 
behavior of the program participants or the program was not 
implemented sufficiently to bring about the desired change in 
the behavior of the program participants. If the activities 
related to the objective of this vocational education program 
were organized in the format of an activity analysis table, 
the percent completion of activities to date would provide a 
measure of the degree to which the activities related to this 
objective were implemented. On the basis of the percent com- 
pletion to date, the outcome could be attributed to the inef- 
fectiveness of the vocational services provided or the failure 
of the program to sufficiently implement the services related 
to that objective. 

In summary, activity analysis is an approach to improving 
the evaluation design of ongoing educational programs which 
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facilitates communication at several levels. Initially, it 
enables the program administrator and his staff to communicate 
the activities and intents of the program to the evaluation 
consultant so that a comprehensive evaluation design for the 
program can be developed efficiently. During the process of 
formulating this design, the activity analysis tables facili- 
tate communication between the program staff and evaluation 
consultant. Finally, activity analysis provides a measure 
of the status of implementation of the program which communi- 
cates to the funding agency the degree to which _Uiej>rogram 
activities related to each objective have been implemented. 
Increased communication in these areas through the applica- 
tion of activity analysis is a positive step toward improving 
the evaluation design, and subsequently, the overall quality 
of evaluation within an ongoing educational program. 



